Papers with word segmentation system
Urdu Word Segmentation using Conditional Random Fields (CRFs) (C18-1)
Copied to clipboard
| Challenge: | Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission and space insertion issues. |
| Approach: | They propose a word segmentation system for Urdu which uses a Conditional Random Field sequence modeler with orthographic, linguistic and morphological features. |
| Outcome: | The proposed system achieves an F1 score for word boundary identification and 0.85 for sub-word boundary identification tasks. |
Syllable-based Neural Thai Word Segmentation (2020.coling-main)
Copied to clipboard
| Challenge: | Word segmentation is a challenging pre-processing step for Thai Natural Language Processing due to the lack of explicit word boundaries. |
| Approach: | They propose a neural Thai word segmenter that uses syllable embeddings to capture linguistic constraints and dilated CNN filters to capture the environment of each character. |
| Outcome: | The proposed system outperforms the previous state-of-the-art system on both in-domain and out-domain datasets. |